Over the last decade, malicious software has become a pervasive problem for Internet users as many networked resources include vulnerabilities that are susceptible to attack. For instance, over the past few years, more and more vulnerabilities are being discovered in software that is loaded onto network devices, such as vulnerabilities within operating systems for example. While some vulnerabilities continue to be addressed through software patches, prior to the release of such software patches, network devices will continue to be targeted for attack by exploits, namely information such as computer code that attempts to take advantage of a vulnerability in computer software by acquiring sensitive information or adversely influencing or attacking normal operations of the network device or the entire enterprise network.
In particular, a technique known as “heap spraying” allows an exploit residing within software of a network device to more easily take advantage of a vulnerability within software (e.g., an application or an operating system) running on the network device. During a heap spray attack, an exploit inserts a portion of code including a sequence of No Operation (NOP) instructions and a sequence of bytes at a particular location within memory of the network device. The sequence of bytes may be directed at performing malicious activity, wherein the sequence of bytes may be in the form of code, commonly referred to as “shellcode.” This particular location is a predictable memory location within a heap (e.g., a predetermined amount of virtual memory allocated for the software). Once code has been inserted into the heap, the exploit may be triggered during processing which causes the application to execute the shellcode thereby allowing the exploit to assume control of the execution of code on the network device. This enables the attacker to execute instructions on the network device which may be malicious to the network device, to its content or to the owner of the network device. Heap spray attacks use chunks of No Operation (NOP) instructions, also known as NOP sleds, to facilitate the execution of the shellcode. By orienting a NOP sled at an address just prior to shellcode, the execution flow of the processing unit of the network device is quickly directed to the shellcode.
The NOP instructions may be a sequence of known instructions (e.g., “patterns”). Conventional heap spray detection methods attempt to identify heap spray attacks by detecting the NOP sleds and comparing the sequence of NOP instructions to NOP instructions appearing on a list of permitted (and non-malicious) NOP instructions. Such a list may be referred to as a “whitelist” of NOP instructions. Those sequences of NOP instructions found on the whitelist are dismissed as non-malicious. However, those sequences of NOP instructions found in the heap but not found on the whitelist are flagged as a probable NOP sled portion of a heap spray attack.
Several issues with a detection strategy using a whitelist of NOP instructions exist. First, comparing sequences of NOP instructions to NOP instructions appearing on a whitelist results in a high number of false positives as a whitelist rarely contains all possible permitted or non-malicious NOP instructions. Several non-malicious NOP instructions not appearing on the whitelist may reside in received network traffic. This in turn prompts the detection system to improperly flag one or more objects within the network traffic as containing or, viewed differently, instigating, when processed, a heap spray attack and return a false positive to a network administrator.
Second, as false positives are reported, the NOP instruction that caused the false positive is typically added to the whitelist in order to prevent future false positives based on that particular non-malicious NOP instruction. However, a whitelist of all possible non-malicious NOP instructions is untenable because such as list would require constant updating.